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Outline 


• Reexamine SCSI and SAN. 

• Why do we need iSCSI? 

• What is iSCSI? 

• Go deeper - examine the iSCSI protocol. 

• Live example - UNH iSCSI reference implementation 

• Nothing is perfect: iSCSI limitation. 
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SCSI Protocol 


SCSI - Small Computer System Interface; 

SCSI became an ANSI standard in 1986; 

SCSI is both a bus hardware specification and a 
command set; 

SCSI makes computer systems complete device 
independent. 

SCSI has high speed range from IMB/s - 320MB/s 



Common words in SCSI 


• SCSI Initiators are the SCSI devices that start the I/O 
process and SCSI Targets are the SCSI devices that 
respond to a request to perform an I/O process. 

• CDB - Command Descriptor Block 

• LUN - Logical Units (max 8 LUNs per device) 

• LB A - Logical Block Address 

- A linear address space mode to mask the details. 
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8 SCSI Command Phases 


• BUS FREE, ARBITRATION, SELECTION, 
RESELECTION, 

• Information Transfer Phases: 

- COMMAND, DATA, STATUS, MESSAGE; 



COMMAND 



STATUS 



^- 

MESSAGE IN 



INITIATOR 


MESSAGE OUT 


TARGET 



DATA IN 





DATA OUT 
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SCSI Commands 


• Several important SCSI commands: 

- TEST UNIT READY, 

- READ CAPACITY, 

- READ 6, 10, 12, 

- WRITE 6, 10, 12, 

- REQUEST SENSE. 


Bit 

Byte 


0 

Operation Code (08h) 

1 

Logical Unit Number | Logical Block Address (MSB) 

2 

Logical Block Address 

3 

Logical Block Address 

4 

Transfer Length 

| 5 

Vendor Unique | Reserved | NACA | Obsolete | Link 










SAN vs. (Direct-attached) SCSI 


Any to any; 

Consolidation; 

Availability 

Scalability 

Bandwidth 


Clients Server 



Sample LAN Configuration 


Servers 



Sample SAN Configuration 






















Why do we need iSCSI? 


Limitation of SCSI 

- Point to point; 

- # of devices (8/16 or 7/15) and # of LUNs 

- Cable length; (25m in SCSI or 12m in Ultra SCSI) 

Limitation of FC-S AN 

- Interoperability issues and incompatibility with existing 
infrastructure; 

- Interoperability issues and incompatibility among different 
FC-SAN from different vendors; 

- Very expensive to install and maintain; 



What is iSCSI? 


• iSCSI - Internet SCSI protocol; 

• A mapping of the SCSI over the TCP protocol. 


user 

User 

Application 

Application 










Important Concepts in iSCSI 


• Connection - A connection is a TCP connection. Communication 
and target occurs over one or more TCP connections. The TCP 
connections carry control messages, SCSI commands, parameters, 
and data within iSCSI Protocol Data Units (iSCSI PDUs). 

• Session - The group of TCP connections that link an initiator with 
a session (loosely equivalent to a SCSI I-T nexus). TCP 
connections can be added and removed from a session. Across all 
connections within a session, an initiator sees one and the same 
target. 

• Command - iSCSI Commands, SCSI Commands. 

• CID, SSID, CmdSN. 
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More about iSCSI Session/Connection 


Different session/connection phases 

- Login Phase 

• Creates a TCP connection; 

• Authenticates each party; 

• Negotiates operational parameters; 

• Creates or marks connection to a session; 

- Full Feature Phase 

• Data transfer 

Different session types: 

- Normal 

- Discovery 



iSCSI Initiator Connection State Diagram 


T13 

• SI: FREE 

• S2:XPT_WAIT 

• S4: IN_LOGIN 
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iSCSI Target Connection State Diagram 
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iSCSI Session State Diagram 
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Negotiable Parameters 


Security parameters and operational parameters; 
“key=<value>” pairs; 

Some important operational parameters: 

- SessionType, 

- MaxConnections and MaxOutstandingR2T; 

- InitialR2T and ImmediateData; 

- MaxRecvDataSegmentLength, MaxRecvPDULength, 
DataPDULength,.. 



iSCSI PDU 


• PDU (Protocol Data Unit): 
The initiator and target divide 
their communications into 
messages. The term "iSCSI 
protocol data unit" (iSCSI 
PDU) is used for these 
messages. 


Byte/ 0 | 1 | 2 | 3 | 

/ I I I I 

10 1 2 3 4 5 6 710 1 2 3 4 5 6 710 1 2 3 4 5 6 710 1 2 3 4 5 6 71 

+-+-+-+-+ 

0/ Basic Header Segment (BHS) / 

+/ / 

+ - + - + -+-+ 

48/ Additional Header Segment 1 (AHS) (optional) / 

+/ / 

+-h-1-1-h 

/ Additional Header Segment 2 (AHS) (optional) / 

+/ / 


+ - + - + - +■ 


+ - + - + - +- 

/ Additional Header Segment n (AHS) (optional) 

+/ 

+ - + - + - +- 


/ 

/ 


+-+-+-+-+ 

k/ Header-Digest (optional) / 

+/ / 

+ - + - +-+-+ 

1/ Data Segment(optional) / 

+/ / 

+ - + - + - +-+ 

m/ Data-Digest (optional) / 

+/ / 

+ - + -+- + - + 


1U 













iSCSI Basic Header Segment 


Byte/ 0 | 1 | 2 

/ 1 1 

10 1 2 34 5 6 710 1 2 34 5 6 710 1 2 34 5 

1 3 

1 

67|01234! 

5 6 71 

0|.|I| Opcode |F| Opcode-specific fields 



4|TotalAHSLength | DataSegmentLength | 

8| LUN or Opcode-specific fields 

121 

+-+-+- 

-+- 

1 

+ 

1 

-+ 


16| Initiator Task Tag 


20/ Opcode-specific fields / 

+/ / 


48 
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SCSI Command PDU 


Byte/ 0 | 1 | 2 | 3 | 

/ I I I I 

|0 123456710 123456 7|0 123456 7|01234567| 


0|.|I| 0x01 |F|R|W|. .|ATTR | Reserved 

+-+-+- 

4|TotalAHSLength | DataSegmentLength 


I 


8| Logical Unit Number (LUN) 

12! 


16| Initiator Task Tag 

+-+- 

20| Expected Data Transfer Length 


24| CmdSN 


28| ExpStatSN 


32/ SCSI Command Descriptor Block (CDB) 

+/ 


48/ AHS (Optional) 

-i-f- 

x/ Header Digest (Optional) 


y/ (DataSegment, Command Data) (Optional) 
+/ 


z/ Data Digest (Optional) 
















SCSI Response PDU 

Byte/ 0 | 1 | 2 | 3 | 

/ I I I I 

|0 123456710 123456 7|0 123456 7|01234567| 


0|.|.| 0x21 |1|. .|o|u|0|U|.| Response 

+-+-+- 

4|TotalAHSLength | DataSegmentLength 


| Status 


I 


8| Reserved 
+ 

12 | 


16| Initiator Task Tag 

+-+- 

20| SNACK Tag or Reserved 


24| StatSN 

+ - 

28| ExpCmdSN 


32| MaxCmdSN 

+ - + - +- 

36| ExpDataSN or Reserved 

+-+-+- 

40| Bidirectional Read Residual Count or Reserved 

-l-1-1- 

44| Residual Count or Reserved 

-i-1-.(- 

48| Header-Digest (Optional) 


/ Data Segment (Optional) 


| Data-Digest (Optional) 
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Simple Read 


-+— 

Initiator Function| 

PDU Type 

1 

Target Function 

Command request 
(read) 

ISCSI Command (READ)»> 

1 

1 

1 



1 


|Prepare Data Transfer 

Receive Data 

1 

«< SCSI Data-in 

1 

Send Data 

Receive Data 

1 

«< SCSI Data-in 

1 

Send Data 

Receive Data 

1 

«< SCSI Data-in 

1 

Send Data 


1 

«< SCSI Response 

I Send Status and Sense 

Command Complete 

1 


1 

-+- 
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Simple Write 


|Initiator Function| PDU Type | Target Function | 

H-1-H-1- 

| Command request |SCSI Command (WRITE)»>| Receive command | 

| (write) | | and queue it | 

+-+-+-+ 

| | | Process old commands| 

+-+-+-+ 

| | | Ready to process | 

j | <« R2T I WRITE command | 

+ - + - + - + 

| Send Data | SCSI Data-out »> | Receive Data | 

+ - + - + - + 

| | <« R2T | Ready for data | 

+ - + - + - + 

| | «< R2T | Ready for data | 

+-+-+-+ 

| Send Data | SCSI Data-out >» | Receive Data | 

+ - + - + - + 

| Send Data | SCSI Data-out »> | Receive Data | 

+-H-1-h 

| | «< SCSI Response |Send Status and Sensei 

•I-1-—-—-1-—-h 

| Command Complete | | | 
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Read with DataSN 


|Initiator Function| PDU Type I Target Function I 


| Command request | SCSI Command (READ)»> | | 

I (read) | | | 


Receive Data | <« SCSI Data-in 

| DataSN =0, F=0 


| Receive Data | «< SCSI Data-in 

| | DataSN = 1, F=0 


| Receive Data | «< SCSI Data-in 

| | DataSN =2, F=1 


| | «< SCSI Response 

j | ExpDataSN = 3 


| Command Complete | 


| Prepare Data Transfer| 

+ - + 

| Send Data | 


| Send Data | 


| Send Data | 


|Send Status and Sense | 
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Write with DataSN 


-+— 

Initiator Function| 

PDU Type & Content 

1 

Target Function 

Command request 
(write) 

ISCSI Command (WRITE)»>| 

1 1 

Receive command 
and queue it 

| | Process old commands 


1 

1 

«< R2T 

R2TSN = 0 

1 

1 

Ready for data 


1 

1 

«< R2T 

R2TSN = 1 

1 

1 

Ready for more data 

Send Data 
for R2TSN 0 

1 

1 

SCSI Data-out »> 
DataSN = 0, F=0 

1 

1 

Receive Data 

Send Data 
for R2TSN 0 

1 

1 

SCSI Data-out »> 
DataSN =1, F=1 

1 

1 

Receive Data 

Send Data 
for R2TSN 1 

1 

1 

SCSI Data »> 

DataSN = 0, F=1 

1 

1 

Receive Data 


1 

1 

«< SCSI Response 
ExpDataSN = 0 

|Send Status and Sense 

1 

Command Complete 

1 


1 

-+- 
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Bidirectional DataSN 


|Initiator Function| PDU Type 


| Command request |SCSI Command »> 
j (Read-Write) | Read-Write 


| <« R2T 

| R2TSN = 0 


| * Receive Data | «< SCSI Data-in 

| | DataSN =0, F=0 


| * Receive Data | «< SCSI Data-in 

| | DataSN =1, F=1 


| * Send Data | SCSI Data-out »> 

| for R2TSN 0 | DataSN =0, F=1 


| | «< SCSI Response 

j | ExpDataSN = 2 

+ - + - 

| Command Complete | 


| Target Function | 


| Process old commands | 


| Ready to process | 

| WRITE command j 


| Send Data 


| Send Data 


| Receive Data | 


|Send Status and Sense | 













Unsolicited and Immediate Write 


+ - + - +-+ 

|Initiator Function| PDU Type & Content | Target Function | 


| Command request | SCSI Command (WRITE)»>| Receive command | 

| (write) |F=0 I and data j 

|+ immediate data | | and queue it | 

+-+-+-+ 

| Send Unsolicited | SCSI Write Data »> | Receive more Data | 

| Data | DataSN = 0, F=1 | | 

+-+-+-+ 

| | | Process old commands | 


| | «< R2T | Ready for more data | 

| | R2TSN = 0 | | 

+-+-+-+ 

| Send Data | SCSI Write Data »> | Receive Data | 

| for R2TSN 0 | DataSN = 0, F=1 | | 

+ - + - +-+ 

| | «< SCSI Response |Send Status and Sense | 

II I I 

+ - +-+-+ 

| Command Complete | | | 


25 












PDU Trace of a Read Request 


A READ request: 32768 bytes of data (64 * 512 bytes) starting LB A = 18572, ending 
LB A = 18635. Current initiator's CmdSN = 22264. Current target's StatSN counter is 
67. MaxRecvPDULength = 12288, so we have 3 separate Dataln PDUs having DataSN 
numbers 0, 1,2: the first 12288 bytes; the second 12288 bytes; the last 8192 bytes. 
DataPDUInOrder = yes. 

I->T: opcode = 0x01, F = 1, R = 1, DSL = 0, ITT = 88931, EDTL = 32768, CmdSN = 
22264, ExpStatSN = 67, CDBopcode = 0x28, CDBlba = 18572, CDBlength = 64; 

T->I: opcode = 0x25,1 = 1, F = 0, DSL = 12288, ITT = 88931, ExpCmdSN = 22265, 
DataSN = 0, BufferOffset = 0; 

T->I: opcode = 0x25,1 = 1, F = 0, DSL = 12288, ITT = 88931, ExpCmdSN = 22265, 
DataSN = 1, BufferOffset = 12288; 

T->I: opcode = 0x25,1 = 1, F = 0, DSL = 8192, ITT = 88931, ExpCmdSN = 22265, 
DataSN = 2, BufferOffset = 24576; 

T->I: opcode = 0x21,1 = 1, F = 1, Response = 0, Status = 0, DSL = 0, ITT = 88931, 
StatSN = 67, ExpCmdSN = 22265; 



UNH iSCSI Reference Implementation 


• Two-layer structure 

- SCSI middle layer (SIML, STML) 

• the common portions of what these Initiators and Targets need to do 
in terms of a logical unit of code that is responsible for processing 
SCSI commands, data and responses. 

- FED (FETD) 

• A relatively simpler front-end driver can be written that handles the 
details of the SCSI Transport Protocol itself. 


27 



UNH Implementation Overview 


SCSI Initiator SCSI Target 



SCSI Commands/Data 
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SCSI Disk IDE Disk 






































SML 


UNH Initiator Host Interface 


_ done 

scsi unreqister 


scsi register 


















UNH Initiator Code 












To Initiator 


UNH Target Code Piece (READ) 



SCSI Target Thread 

















From Initiator From Initiator To Initiator 




















UNH Target Storage Devices 


• UNH code supports 3 storage 
modes: MEMORYIO, 
FILEIO, DISKIO; 

• DISKIO mode can choose to 
operate on 3 different layers; 





SCSI Disk 










Limitation of iSCSI 


TCP/IP protocol overhead 

- Performance 

- Qos 

Network Latency 
Protocol overhead 



Conclusion 


• iSCSI is... 
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Reference & More Information 


Most of the stuff in this presentation can be found from 

- iSCSI Standard: 

• http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iscsi-20.txt 

- University of New Hampshire Implementation: 

• Http://www.iol.unh.edu/consortiums/iscsi/ 

Feel free to contact Ming Zhang (mingz@ele.uri.edu) if 
you have any question about the content in this 
presentation. 


